MML is not consistent for Neyman-Scott

Minimum Message Length (MML) is a popular method for statistical inference, belonging to the Minimum Description Length (MDL) family. It is a general name for any of several computationally-feasible approximations to the generally NP-Hard Strict Minimum Message Length (SMML) estimator. One often-cited showcase for the power of MML is the Neyman-Scott estimation problem, where most popular estimation algorithms fail to produce a consistent result. MML’s performance on the Neyman-Scott problem was analysed by Dowe and Wallace (1997) and by Wallace (2005) and MML was shown to be consistent for the problem. However, this analysis was not performed on SMML, but rather on two SMML approximations: Wallace-Freeman and Ideal Group. As for most estimation problems of more than a single dimension, the exact SMML solution is not known for Neyman-Scott. Our contributions: 1. We analyse the Dowe-Wallace solution, and show that it hinges critically on the use of an unnatural prior for the problem. We argue that the Jeffreys prior is a more natural prior to assume in this case. 2. Re-analysing the problem over its Jeffreys prior, we show that both the Ideal Group and the Wallace-Freeman approximations converge to the (inconsistent) Maximum Likelihood (ML) solution. The Wallace-Freeman MML approximation is perhaps the most frequently used MML flavour. 3. We develop novel techniques that enable determining properties of the SMML estimator for some general families of estimation problems without requiring a full construction of the estimator. 4. Using these tools, we show that the SMML estimator for many problems, including Neyman-Scott, is not a point-estimator at all. Rather, it maps each observation to an entire continuum of estimates. 5. We show that for every set of Neyman-Scott parameters, as well as in the asymptotic cases, the SMML estimate for each observation includes the Maximum Likelihood estimate. 6. In particular, we prove in this way that SMML is not consistent for this problem. The importance of this is that it demonstrates that the inconsistency is not only in Email address: [email protected] (Michael Brand) Preprint submitted to TBD October 17, 2016 the Ideal Group or any other MML approximation, but rather appears already in the strict form of MML, and therefore reflects properties of the MML framework itself. 7. We uncover some hidden connections between SMML, Wallace-Freeman MML and ML, and describe families of estimation problems for which all three coincide. 8. We discuss methodological problems in the arguments put forward by previous authors, who argued that MML is consistent, and specifically that it is consistent for the Neyman-Scott problem.